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1.  INTRODUCTION 


Veterans  present  to  primary  care  providers  (PCPs)  with  posttraumatic  stress  (PTS)  symptoms 
because  many  are  resistant  to  specialty  mental  health  care.  Most  PCPs  have  not  been  trained  to 
assess  for  and  manage  symptoms  of  PTS  or  motivate  Veterans  to  engage  in  treatment.  This  can 
result  in  missed  opportunities  to  intervene  to  prevent  chronic  mental  and  physical  health  problems. 
The  project  aims  are  to:  (1)  iteratively  design  a  new  web-based  PTS  and  Motivational  Interviewing 
training  for  PCPs  using  Virtual  World  technology  to  enhance  interactivity;  (2)  implement  a  robust 
evaluation  including  a  randomized  control  trial  for  clinically  valid  outcome  measurement;  (3)  conduct  a 
summative  evaluation  to  inform  national  “scale-up”  dissemination  and  implementation.  The  project  will 
produce  a  deliverable  that  will  improve  access  to  quality  clinical  care  for  our  warfighters  suffering  with 
PTS.  This  report  shares  progress  made  during  Year  2  of  the  project. 


2.  KEYWORDS 

Virtual  world;  PTSD;  medical  education;  virtual  training;  curriculum  development;  motivational 
interviewing 
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3.  ACCOMPLISHMENTS 


Year  1  of  the  project  was  dedicated  to  the  developmental  formative  evaluation  and  curriculum 
development  for  the  Virtual  World  PTSD  and  Motivational  Interviewing  training.  The  Principal 
Investigator,  Project  Coordinator,  and  other  Co-Investigators  advanced  the  development  of  the 
training  curriculum,  focusing  on  building  content  that  is  consistent  with  the  overall  learning  objectives. 
Concurrently,  the  vendors  at  Heyden  Ty  provided  guidance  for  virtualizing  elements  of  the  curriculum 
to  build  an  engaging  and  impactful  training. 

Year  2  of  the  project  was  largely  dedicated  to  building  the  virtual  world.  The  curriculum  and  content 
developed  during  Year  1 ,  were  implemented  into  the  learning  environment  by  our  developers  at 
Chant  Newall  Development  Group  (CNDG).  As  the  virtual  world  was  being  built,  the  team  continued 
to  refine  the  curriculum  and  content  to  ensure  a  high  quality  educational  product  that  achieves 
learning  objectives  and  fits  within  the  limitations  of  the  virtual  world  platform.  The  research  team, 
vendors  at  HeydenTy,  and  developers  at  CNDG  met  weekly  to  go  over  the  progress  of  the  build  and 
project  deadlines. 

Furthermore,  we  conducted  a  focus  group  with  Co-Investigators  to  get  feedback  on  the  training 
curriculum  and  activities.  The  Co-Investigators  range  from  Primary  Care  providers  to  Mental  Health 
providers,  giving  well  rounded  feedback.  The  feedback  from  the  focus  group  was  implemented  back 
in  to  virtual  world  learning  environment.  We  recruited  for  the  next  focus  group  that  includes  providers 
who  are  unrelated  and  unfamiliar  with  the  project.  The  focus  group  was  conducted  on  October  18, 

201 7.  These  individuals  went  through  a  demo  training  in-world  and  provided  feedback  which  we  are 
implementing  in  to  the  learning  environment. 

The  Project  Coordinator  continues  to  work  on  the  control  condition  (the  online  training  that  was  tested 
in  our  prior  study).  The  slides  have  been  updated  and  we  are  in  the  process  of  adding  interactive 
content.  Because  the  control  is  a  video,  we  must  re-record  some  parts  of  the  video  to  be  consistent 
with  the  DSM-5  updates.  We  have  identified  who  will  be  the  voice  on  the  video  and  are  in  the  process 
of  re-rerecording.  The  Project  Coordinator  has  also  started  communicated  with  universities  regarding 
CME  accreditation.  Both  arms  need  to  be  submitted  for  approval  in  the  CME  application  process.  We 
estimate  that  both  tasks  will  be  complete  in  November  2017. 


a.  What  were  the  major  goals  of  the  project? 


Timeline 

(Months) 

Completed 

Major  Task  1:  Obtain  local  IRB  and  VA  R&D  and  HRPO  Approvals. 

1-2 

1 

Subtask  1:  File  protocol  with  Local  IRB 

1 

1 

Subtask  2:  File  protocol  with  VA  R&D 

1 

1 

Subtask  3:  File  protocol  with  FIRPO 

1-2 

1 

Subtask  4:  Make  any  required  revisions  and  resubmit  in  the  above  order. 

1-2 

3 

Subtask  5:  Obtain  Local  IRB/  VA  R&D/  FIRPO  Approval 

2 

1 
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Milestone  #1:  Local  IRB/  VA  R&D/  HRPO  Approval 

1 

Aim  1:  To  conduct  a  developmental  formative  evaluation  to  iteratively 
inform  a  Virtual  World  (VW)  design  based  on  our  prior  web-based 
posttraumatic  stress  (PTS)  training  for  primary  care  providers  (PCPs). 

Major  Task  2:  Semi-Structured  interviews  with  project  stakeholders/key 
informants  to  inform  curriculum  content  and  instructional  design 

3 

7-10 

Subtask  1:  Recruit  &  enroll  stakeholders/key  informants 

3 

7-9 

Subtask  2:  Qualitative  analysis  of  curriculum  content  and  instructional  design. 

3 

9-12 

Major  Task  3:  Begin  Virtual  World  (VW)  build 

2-12 

7-19 

Subtask  1:  Design  curriculum  based  on  data  from  semi-structured  interviews. 

2-4 

8-12 

Subtask  2:  Host  VW  learning  environment  (initially  in  Second  Life) 

5 

7 

Subtask  3:  Build  an  Orientation  Center 

6 

7 

Subtask  4:  Create  a  storyboard 

7 

9 

Subtask  5:  Import  and  create  virtual  objects 

9 

9 

Subtask  6:  Create  avatar  types 

9 

9 

Subtask  7:  Secure  the  VW  environment 

10 

In  progress 

Subtask  8:  Conduct  quality  checks 

11 

In  progress 

Subtask  9:  Migrate  to  other  VW  platform  (e.g.  UNITY) 

12 

17 

Major  Task  4:  Independent  review  of  new  VW  training  using  a  focus  group 

12-15 

24-25 

Subtask  1:  Recruit  stakeholders/key  informants,  including  PCPs,  who  were 
not  involved  in  initial  build  recommendations 

12-14 

24 

Subtask  2:  Conduct  1  Focus  Group 

13,  14 

25 

Subtask  3:  Make  revisions  based  on  feedback 

15 

25 

Major  Task  5:  Refinement  of  the  VW  training 

16 

24-  25 

Subtask  1:  Recruit  independent  reviewers  for  Focus  Group 

16 

21 

Subtask  2:  Conduct  final  focus  group  to  review  revisions 

16 

22 

Subtask  3:  Study  co-investigators  beta-test  training  and  project  staff 

16 

In  progress 

Major  Task  6:  Refinement  of  prior  online  training  (Control)  to  make  it  a 
more  apt  comparison  for  RCT. 

17 

17-18 

Subtask  1:  Update  content  to  be  consistent  with  PTS  diagnosis  in  DSM  5 

17 

20 

Subtask  2:  Add  simple  interactive  content 

17 

In  progress 

Milestone  #2:  VW  and  Control  condition  complete 

Incomplete,  est.  26 
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Specific  Aim  2:  Conduct  a  randomized  controlled  trial  of  a  Virtual  World 
training  vs.  a  traditional  web-based  training  to  evaluate  effectiveness  using 
"gold  standard"  educational  outcomes. 

Major  Task  7:  Obtain  CME  accreditation  for  the  trainings 

18 

In  progress 

Subtask  1:  Contact  Universities'  CME  offices  and  register  as  a  vendor 

18 

23 

Subtask  2:  Collaborate  with  CME  offices  to  host  training 

18 

In  progress 

Major  Task  8:  Develop  self-report  measure  battery  to  assess  relative  change 
in  self-reported  PTS-related  knowledge  and  clinical  skills  self-efficacy. 

18 

23-  present 

Subtask  1:  Develop  test  of  self-reported  PTS-related  knowledge  and  clinical 
skills  self-efficacy 

18 

23 

Subtask  2:  Beta  test  online  hosting  of  self-report  measures 

18 

Incomplete 

Major  Task  9:  Develop  telephone  Standardized  Patient  (SP)  Interview  to 
assess  relative  change  in  PTS-related  competency  (Primary  Outcome)  and 

Ml  skills  to  assess  and  manage  PTS  symptoms  and  to  motivate  engagement 
in  care. 

18-20 

Incomplete 

Subtask  1:  Train  the  SP  actors  and  calibrate  fidelity  monitoring. 

18,  19 

Incomplete 

Subtask  2:  Calibrate  the  SP  assessment  instrument 

20 

Incomplete 

Subtask  3:  Calibrate  the  use  of  the  MITI  global  rating  of  Ml  performance 

20 

Incomplete 

Major  Task  10:  Recruitment/Eligibility  Screening 

19 

In  progress 

Subtask  1:  Develop  and  Refine  eligibility  screening 

19 

In  progress 

Subtask  2:  Beta  test  eligibility  link  to  minimize  false  eligibility 

19 

Incomplete 

Major  Task  11:  Recruit  PCPs  from  VA,  DoD,  and  community 

20-39 

In  progress 

Subtask  1:  Develop  recruitment  tools  (email,  e-flyers) 

20 

In  progress 

Subtask  2:  Send  email  blasts 

20-35 

Incomplete 

Major  Task  12:  Conduct  Enrollment  and  Informed  Consent 

19-39 

Incomplete 

Subtask  1:  Verify  eligibility  requirements 

19-35 

Incomplete 

Subtask  2:  Assign  subject  ID 

19-35 

Incomplete 

Major  Task  13:  Conduct  baseline  assessments 

19 

Incomplete 

Subtask  1:  Disseminate  baseline  link  to  eligible  participants 

19 

Incomplete 

Major  Task  14:  Obtain  Patient  Outcomes 

25-39 

Incomplete 

Subtask  1:  Make  VA  data  request  and  clean  administrative  data 

25-39 

Incomplete 

Subtask  2:  Obtain  and  analyze  VA  outcomes  data  on  patient  health  services 
utilization 

25-39 

Incomplete 

Major  Task  #15:  Randomization 

19-39 

Incomplete 

Subtask  1:  Develop  stratified  block  randomization  list 

19 

Incomplete 

Major  Task  #16:  VW  vs.  Control  training 

19-39 

Incomplete 

Subtask  1:  Conduct  3  VW  trainings 

Over  21 

months 
(1  training 
every  7 
months) 

Incomplete 

Subtask  2:  Control  training 

Ongoing 
for  21 

months 

Incomplete 
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What  was  accomplished  under  these  goals? 

Major  Task  1 :  Obtain  local  IRB  and  VA  R&D  and  HRPO  Approvals.  -  Complete 

Subtask  1 :  File  protocol  with  Local  IRB. 

Subtask  2:  File  protocol  with  VA  R&D. 

Subtask  3:  File  protocol  with  HRPO. 

Subtask  4:  Make  any  required  revisions  and  resubmit  in  the  above  order. 

Subtask  5:  Obtain  Local  IRB/  VA  R&D/  HRPO  Approval. 

Submitted  to  and  Approved  by: 

University  of  California  San  Francisco  Committee  on  Human  Research-  14-15004 

•  Study  Approval  ( Continuing  Review)  09/03/20 1 6-05/04/20 1 7 

•  Amendment  1:  personnel  changes  -  08/12/2015 

•  Amendment  2:  personnel  changes-  21/12/2015 

•  Amendment  3:  personnel  changes,  increased  number  of  SSI  Vs,  interview  guide-  09/03/20 1 ) 

•  Protocol  Violation:  acknowledged  on  09/27/2016-  Semi-structured  interviews  were  conducted  as 
part  of  the  study's  developmental  formative  evaluation.  The  subjects  (interviewees)  are  primary 
care  providers  at  the  VA.  The  protocol  states  that  the  interviews  are  to  be  conducted  via 
telephone.  On  July  13,  2016,  immediately  prior  to  the  interview,  the  subject  requested  to  be 
interviewed  in  person,  rather  than  over  the  phone.  This  was  a  non-veteran  subject,  a  physician 
on  staff  at  SFVAMC.  The  researcher  complied  with  the  subject's  request  and  conducted  the 
interview  in  person.  The  verbal  consent  process  was  executed  as  part  of  the  interview,  however, 
no  signed  consent  was  obtained  for  this  incident. 

•  Continuing  Review:  with  personnel  changes-  approved  22/03/20 1 7 

•  Amendment  4:  personnel  changes-  12/05/17 

•  Amendment  5:  personnel  changes;  administrative  changes  to  the  Information  Sheet; 
modification  to  the  recruitment  material;  addition  of  a  new  Interview  Guide-  09/06/2017 

•  Amendment  6:  participant  thank  you  letter-  30/8/20 1 7 

•  Amendment  7:  personnel  changes,  recruitment  document  changes,  website  text,  interview 
guide-  13/10/17 

VA  Research  &  Development  and  Clinical  Research  Workgroup 

•  CRW- Approval  22/04/2015 

•  VA  R&D  -  Approval  07/05/2015 

o  Renewed  09/03/2016 
o  Renewed  22/03/2017 

DoD  Human  Research  Protection  Office  A1 8590 

•  Approval  1 7/9/20 1 5 

•  Continuing  Review  Approved  26/04/20 1 6 

•  Continuing  Review  Approved  18/04/2017 


Major  Task  2:  Semi-Structured  interviews  with  project  stakeholders/key  informants  to  inform 
curriculum  content  and  instructional  design  -  Complete 
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•  The  qualitative  team  began  interviewing  1 1  Clinical  Educators  and  healthcare  leadership. 
Additional  interviews  may  be  conducted  with  educational  experts  and  Information  Technology 
(IT)  experts  on  an  ongoing  basis.  The  Qualitative  Researchers  assert  that  this  will  produce  the 
most  relevant  data,  since  there  will  be  beta  elements  of  the  training  to  share  and  discuss  with 
interviewees  at  different  milestones  of  the  curriculum  and  virtual  training  development. 

•  The  overall  conclusions  from  the  developmental  formative  evaluation  include: 

1 .  Findings  about  challenges  in  primary  care  practice  related  to  providing  care  to  veterans 
with  PTSD  as  well  as  treatment  choices  by  PCPs  and  use  of  motivational  interviewing  are 
consistent  with  the  published  studies. 

2.  Findings  support  the  need  for  training  for  PCPs  focused  on  PTSD  and  Ml  applied  to 
providing  care  for  patients  with  PTSD. 

3.  Opinions  of  interviewed  PCPs  about  VW  training  were  limited  to  their  insufficient 
understanding  of  VW. 

4.  This  formative  evaluation  generated  implications  for  audience  generation,  content,  and 
training  delivery  and  evaluation,  and  training  generalizability  that  are  being  utilized  for  the 
VW  training  development  in  this  project. 

Subtask  1 :  Recruit  &  enroll  stakeholders/key  informants 

•  Project  Coordinator  and  Research  Assistant  recruited  and  enrolled  stakeholders/key  informants 
from  a  potential  pool  of  participants  established  by  the  PI  and  Project  Team. 

•  Recruitment  began  in  April  2016  and  will  continue  as  deemed  necessary. 

•  PI  and  Qualitative  Researcher  have  conducted  1 1  semi-structured  interviews,  to  date. 

Subtask  2:  Qualitative  analysis  of  curriculum  content  and  instructional  design. 

•  Qualitative  Analyst  analyzed  semi-structured  interview  results  in  relation  to  the  curriculum  content 
and  instructional  design. 

•  Content  experts  from  the  project  team  have  reviewed  and  offered  insight  on  curriculum 
storyboards. 

•  Findings  from  the  analyses  of  ten  interviews  were  shared  in  a  Technical  Report. 


Major  Task  3:  Begin  Virtual  World  (VW)  build  -  In  Progress 

Subtask  1 :  Design  curriculum  based  on  data  from  semi-structured  interviews. 

•  Analyzed  evaluation  data  was  displayed  in  a  matrix  (within  Appendix  A),  and  shared  with  the 
curriculum  development  team  to  inform  the  refinement  of  the  training  curriculum. 

•  Storyboards  detailing  the  content  for  Sessions  1  and  2  of  the  training  were  refined  and  submitted 
to  CNDG.  CNDG  will  transform  the  concepts  into  a  beta  version  of  the  training. 

Subtask  2:  Host  VW  learning  environment  (initially  in  Second  Life,  now  in  Unity) 

•  Project  Team  members  have  been  oriented  to  Unity.  CNDG  hosted  team  members  in  an  existing 
learning  environment  and  solicited  feedback  regarding  the  characteristics  of  virtual  spaces  and 
avatars  to  help  inform  the  environment  they  will  build  specifically  for  this  project  during  a  project¬ 
wide  meeting  on  26-04-2016. 

•  They  submitted  a  video  detailing  their  progress  on  22-06-2016. 

•  CNDG  created  a  virtual  campus,  which  contains  a  hospital  building  with  internal  exam  rooms  for 
Standardized  Patient  interviews. 

•  As  the  developers  at  CNDG  build  additional  training  components,  the  Project  Team  will  continue 
to  monitor  progress  and  offer  feedback. 

Subtask  3:  Build  an  Orientation  Center 

•  CNDG  held  several  orientation  sessions  for  project  staff  and  collaborators  in  the  virtual  world.  The 
Orientation  Center  environment  provides  instructions  on  how  to  navigate,  as  well  as  serves  as  a 
test  for  computer  compatibility. 
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•  Orientation  has  been  completed  for  the  focus  groups  and  is  ready  for  trial  initiation. 

Subtask  4:  Create  a  storyboard 

•  Heyden  Ty,  our  vendor  which  specializes  in  virtual  learning  curriculums,  submitted  storyboards  for 
each  segment  of  the  two-session  training  on  15-06-2016.  These  storyboards  contain  the  proposed 
learner  experience  during  the  various  segments  of  the  training:  Exploratorium,  Didactic/Lecture, 
Small  Group  Standardized  Patient  Interviews,  and  Homework.  Additionally,  they  contain  content 
and  directions  for  the  engineering  experts  at  CNDG. 

•  Much  of  Year  1 ,  Quarter  4  was  spent  refining  these  storyboards  to  include  feedback  from  the 
developmental  formative  evaluation. 

•  The  story  boards  have  been  finalized  by  Dr.  Seal  and  our  consultants. 

Subtask  5:  Import  and  create  virtual  objects 

•  CNDG  created  a  video  which  presents  virtualized  elements  of  the  storyboards  and  curriculum  for 
the  project  team  and  Principal  Investigator  to  review.  Examples  include  polling  stations, 
presentation  screens,  whiteboards  for  instructor  and  learner,  and  click-to-view  information  posters. 
They  will  continue  to  develop  virtual  objects  and  refine  the  environment  until  the  build  is  complete. 

Subtask  6:  Create  avatar  types 

•  CNDG  created  sample  avatars  for  the  training,  based  on  the  specifications  provided  to  them  by 
the  Project  Coordinator.  Additionally,  they  created  a  prototype  of  the  veteran,  which  will  serve  as 
the  case  study  for  the  duration  of  the  training. 

•  CNDG  completed  the  avatar  for  the  character  Alex  and  36  avatars  for  the  trial  participants  to  use 
and  customize. 

Subtask  7:  Secure  the  VW  environment-  In  progress 

•  The  environment  is  nearing  completion,  with  an  estimate  of  being  done  in  month  27  after 
adjusting  for  feedback  based  on  focus  groups  in  month  25.  As  sections  are  completed  they  are 
secured,  such  as  the  orientation  center  and  main  hospital  lobby.  Some  areas  within  the  VW 
continue  to  be  adjusted  for  optimum  user  experience. 

Subtask  8:  Conduct  quality  checks-  In  progress 

•  Quality  checks  are  ongoing.  Different  individuals  on  the  team  routinely  meet  to  test  out 
functionality  of  the  virtual  world  as  it  is  being  developed. 

Subtask  9:  Migrate  to  other  VW  platform  (e.g.  UNITY) 

•  This  subtask  is  no  longer  relevant.  Before  the  build  started,  our  developers  made  the  decision 
to  use  UNITY  through  the  entire  build  process.  The  platform  is  being  built  in  UNITY  and  will  be 
hosted  in  UNITY  so  a  migration  was  never  required. 


Major  Task  4:  Independent  review  of  new  VW  training  using  a  focus  group-  In  progress 
Subtask  1:  Recruit  stakeholders/key  informants,  including  PCPs,  who  were  not  involved  in  initial  build 
recommendations 

•  Recruitment  for  the  focus  group  has  been  completed. 

Subtask  2:  Conduct  1  Focus  Group 

•  Due  to  a  delay  in  finding  a  new  project  coordinator,  there  was  a  delay  in  completing  the  focus 
groups. 

•  The  focus  group  was  completed  on  October  1 8,  201 7. 

Subtask  3:  Make  revisions  based  on  feedback 

•  The  focus  group  was  completed  on  October  1 8,  201 7  so  we  have  not  yet  had  time  to  make 
revisions.  The  qualitative  interviewer  will  compile  the  collected  data  and  present  the  feedback 
for  revisions  in  late  October  2017.  The  revisions  will  be  made  in  November  2017. 
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Major  Task  5:  Refinement  of  the  VW  training-  In  progress 
Subtask  1 :  Recruit  independent  reviewers  for  Focus  Group 

•  The  recruitment  for  the  focus  group  was  completed. 

Subtask  2:  Conduct  final  focus  group  to  review  revisions-in  progress 

•  The  focus  group  was  completed  on  July  1 0,  201 7  with  PCPs  within  the  VA  system. 
Subtask  3:  Study  co-investigators  beta-test  training  and  project 

•  A  meeting  with  the  co-investigators  to  beta-test  the  training  will  we  be  planned  and  occur 
before  the  RCT  starts. 


Major  Task  6:  Refinement  of  prior  online  training  (Control)  to  make  it  a  more  apt  comparison 
for  RCT-  In  progress 

Subtask  1 :  Update  content  to  be  consistent  with  PTS  diagnosis  in  DSM-5 

•  With  the  help  of  the  Psychologist  Co-Investigators  we  have  revised  the  curriculum  to  be 
aligned  with  DSM  5  PTS  criteria. 

Subtask  2:  Add  simple  interactive  content 

•  We  are  working  with  our  web  designer  to  add  some  simple  interactive  content  to  the  Control 
training. 


Major  Task  7:  Obtain  CME  accreditation  for  the  trainings-  In  Progress 
Subtask  1 :  Contact  universities’  CME  offices  and  register  as  a  vendor 

•  Several  universities  have  been  contacted  and  we  will  be  moving  forward  with  CME 
accreditation  at  University  of  California,  San  Francisco  (UCSF).  We  have  a  contact  there  who 
is  ready  to  look  at  our  trainings  when  they  are  complete. 

Subtask  2:  Collaborate  with  CME  offices  to  host  trainings 

•  No  longer  applicable.  VW  will  be  hosted  on  CNDG  controlled  server  allowing  for  easier  access 
by  programmers  in  the  event  of  any  difficulty.  We  will  work  with  UCSF  to  ensure  that  this  set¬ 
up  is  compatible  with  their  systems. 


Major  Task  8:  Develop  self-report  measure  battery  to  assess  relative  change  in  self-reported 

PTS-related  knowledge  and  clinical  skills  self-efficacy-  In  Progress 

Subtask  1 :  Develop  test  of  self-reported  PTS-related  knowledge  and  clinical  skills  self-efficacy 

•  Standardized  outcomes  assessments  have  been  complied. 

•  Assessments  include  outcomes  measures  for  things  such  as  demographics,  clinical  skills  and 
comfort  for  assessing  PTS,  knowledge  of  PTS,  feedback  on  training,  and  intervention  usability. 

Subtask  2:  Beta  test  online  hosting  of  self-report  measures 

•  Our  data  manager  will  input  the  self-report  measures  in  to  our  online  host,  Qualtrics,  in 
November  2017. 

•  Our  project  coordinator  and  data  manager  will  beta  test  the  measures  in  November  201 7. 


Major  Task  9:  Develop  telephone  Standardized  Patient  (SP)  Interview  to  assess  relative 
change  in  PTS-related  competency  (Primary  Outcome)  and  Ml  skills  to  assess  and  manage 
PTS  symptoms  and  to  motivate  engagement  in  care  -  Incomplete 
Subtask  1 :  Train  the  SP  actors  and  calibrate  fidelity  monitoring 

•  This  task  will  be  completed  by  the  time  of  the  start  RCT  in  January  2018. 

•  The  hiring  and  training  takes  of  standardized  patients  will  start  in  November  2017. 
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Subtask  2:  Calibrate  the  SP  assessment  instrument 

•  This  task  will  be  completed  by  the  time  of  the  start  RCT  in  January  201 8. 

•  This  task  will  occur  in  December  2017. 

Subtask  3:  Calibrate  the  use  of  the  MITI  global  rating  of  Ml  performance 

•  This  task  will  be  completed  by  the  time  of  the  start  RCT  in  January  2018. 

•  This  task  will  occur  in  December  2017. 


Major  Task  10:  Recruitment/Eligibility  Screening  -  In  progress 
Subtask  1 :  Develop  and  Refine  eligibility  screening 

•  The  eligibility  screener  has  been  developed. 

•  The  project  coordinator  modified  the  eligibility  screener  and  will  submit  to  the  IRB  for  approval. 
Subtask  2:  Beta  test  eligibility  link  to  minimize  false  eligibility 

•  After  the  changes  to  the  eligibility  screener  have  been  approved  by  the  PI  and  IRB,  the  project 
coordinator  and  data  manager  will  upload  the  screener  to  the  online  platform  and  beta  test  for 
fidelity.  We  estimate  this  will  occur  in  November  201 7. 


Major  Task  1 1 :  Recruit  PCPs  from  VA,  DoD,  and  community  -  In  progress 
Subtask  1 :  Develop  recruitment  tools  (email,  e-flyers) 

•  The  project  coordinator  has  put  together  recruitment  tools  and  submitted  them  to  IRB  for 
approval. 

Subtask  2:  Send  email  blasts 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 


Major  Task  12:  Conduct  Enrollment  and  Informed  Consent  -  Incomplete 
Subtask  1 :  Verify  eligibility  requirements 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 

Subtask  2:  Assign  subject  ID 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 


Major  Task  13:  Conduct  baseline  assessments  -  Incomplete 
Subtask  1:  Disseminate  baseline  link  to  eligible  participants 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 


Major  Task  14:  Obtain  Patient  Outcomes  -  Incomplete 
Subtask  1 :  Make  VA  data  request  and  clean  administrative  data 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 

Subtask  2:  Obtain  and  analyze  VA  outcomes  data  on  patient  health  services  utilization 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 
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Major  Task  #15:  Randomization  -  Incomplete 
Subtask  1 :  Develop  stratified  block  randomization  list 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 


Major  Task  #16:  VW  vs.  Control  training  -  Incomplete 
Subtask  1 :  Conduct  3  VW  trainings 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 

Subtask  2:  Control  training 

•  This  task  is  part  of  the  randomized  control  which  cannot  start  until  major  tasks  1  -1 0  have  been 
completed.  We  expect  the  randomized  control  trial  to  begin  in  January  2018. 


What  opportunities  for  training  and  professional  development  has  the  project 
provided? 

The  providers  who  have  participated  in  the  focus  groups  have  had  the  opportunity  to  learn  more 
about  trainings  in  a  virtual  environment. 


How  were  the  results  disseminated  to  communities  of  interest? 

The  team  presented  an  abstract  at  the  Society  for  Academic  Continuing  Medical  Education  (SACME) 
in  May  2017  in  Scottsdale,  AZ.  We  will  continue  to  create  and  present  products  as  more  data  is 
collected  and  analyzed. 


What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals? 

We  will  continue  working  closely  and  having  weekly  meetings  with  CNDG  and  HeydenTy  to  complete 
the  build  of  the  virtual  world.  The  PI  and  project  coordinator  will  work  closely  to  complete  all  the 
necessary  task  for  the  randomized  control  trial  by  the  end  of  2017,  including,  obtaining  CME 
accreditation,  completing  the  changes  in  the  control  arm  training,  finalizing  the  outcomes 
assessments,  and  training  of  the  Standardized  Patient. 

We  will  start  the  randomized  control  trial  at  the  beginning  of  2018  which  will  include  the  following 
tasks: 

Major  Task  1 1 :  Recruit  PCPs  from  VA,  DoD,  and  community 

Major  Task  12:  Conduct  Enrollment  and  Informed  Consent 

Major  Task  13:  Conduct  baseline  assessments 

Major  Task  14:  Obtain  Patient  Outcomes 

Major  Task  #15:  Randomization 

Major  Task  #16:  VW  vs.  Control  training 
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Major  Task  #17:  Post-training  assessment 

Major  Task  #18:  Follow-up  assessment  after  a  period  of  90  days  of  no  contact 

While,  we  are  currently  behind  schedule  due  to  delays  in  the  build  of  the  virtual  world  and  the 
turnover  of  staff,  we  are  confident  that  we  can  make  up  time  during  the  RCT  and  still  complete  the 
study  on  time.  We  will  accomplish  this  with  aggressive  recruitment  of  subjects  and  shortening  the 
time  intervals  between  intervention  arm  trainings  from  every  7  months  to  every  5-6  months. 
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4.  IMPACT 


What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the 
project? 

While  it  is  too  early  to  cite  specifics,  the  project  has  the  potential  to  make  an  impact  on  primary  care 
provider  education  and  continuing  medical  education  relating  to  health  issues  affecting  both  mind  and 
body,  including  post-traumatic  stress  disorder.  Since  the  training  is  delivered  virtually,  it  can  easily  be 
disseminated  world-wide. 


What  was  the  impact  on  other  disciplines? 

Nothing  to  report. 


What  was  the  impact  on  technology  transfer? 

While  the  product  we  are  developing  is  a  training  for  Primary  Care  Providers,  our  work  has  produced 
a  very  versatile  platform  that  is  now  available  to  the  DOD  and  VA  for  other  purposes.  The  platform  is 
flexible,  extensible,  and  could  easily  be  used  for  other  types  of  trainings  or  gatherings.  Furthermore, 
since  this  is  virtual,  it  could  be  used  within  the  DOD  or  VA  for  providers  around  the  world  to  meet, 
interact  and  easily  disseminate  information  without  costly  travel. 

What  was  the  impact  on  society  beyond  science  and  technology? 

The  developmental  formative  evaluation  and  focus  groups  reveal  a  fair  amount  of  hesitation  toward 
virtual  world  training  for  continuing  medical  education.  Several  PCPs  indicated  generational 
differences  in  acceptability  and  ability  to  navigate  and  learn  in  a  virtual  environment.  As  the  findings 
are  applied  to  the  curriculum  and  design  of  the  training,  the  project  has  the  potential  to  improve  public 
knowledge  and  attitudes  toward  virtual  world  technology  for  the  sake  of  provider  education.  We  are 
developing  this  technology  for  future  healthcare  professionals  so  that  they  will  be  able  to  use  the  next 
generation  of  tools  and  capability.  What  feels  challenging  today  will  feel  like  second-nature  in  the 
future. 
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5.  CHANGES/PROBLEMS 


Changes  in  approach  and  reasons  for  change 

We  originally  proposed  prototyping  and  building  in  SecondLife  and  then  migrating  objects  to  Unity  but 
over  the  period  of  time  from  proposal  to  project  start,  Sinewave  developed  building  tools  and  the 
ability  to  customize  the  viewer  for  our  learners  that  gave  us  much  more  control  and  flexibility.  Given 
the  needs  of  the  DOD  and  the  VA  to  house  the  product  on  their  own  servers,  behind  their  firewall,  we 
opted  to  build  directly  in  Unity  and  use  a  customized  Sinespace  viewer  for  our  learners. 


Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them 

1 .  Resolved:  Our  original  project  coordinator  left  in  December  201 6.  We  were  without  a  project 
coordinator  for  5  months.  In  May  2017,  we  hired  a  new  coordinator  and  unfortunately  he  went  on 
to  pursue  other  opportunities  in  August  2017.  Since  then  an  experienced  project  coordinator,  who 
has  worked  with  Dr.  Seal  for  2.5  years,  has  taken  over  the  project.  The  staff  turnover  resulted  in 
some  delays,  however  we  expect  to  be  back  on  to  schedule  and  the  randomized  control  trial 
portion  of  the  trial  will  still  be  completed  on  time. 

2.  Delay:  As  previously  stated,  there  is  a  delay  in  development  of  the  virtual  world  (major  task  3) 
because  we  are  using  a  newly  developed  platform  (Sinewave  and  Unity)  which  caused  us  to  hit 
unavoidable  glitches  and  bugs  which  we  have  had  to  work  through. 

•  Course  of  action  being  taken:  We  continue  to  work  through  these  bugs  as  they  occur  and  are 
working  very  hard  to  have  the  build  of  the  world  complete  by  December  2017. 

3.  Delay:  We  are  moving  forward  with  the  steps  necessary  to  get  the  randomized  control  portion  of 
the  study  running  by  January  2018.  The  delays  are  largely  due  to  not  being  adequately  staffed  in 
project  year  2.  Now  that  we  have  seasoned  staff  (Project  Coordinator,  Qualitative  Analyst)  on 
board  we  are  very  optimistic  that  we  will  be  able  to  meet  future  milestones.  The  virtual  world  build 
must  be  complete  in  order  to  begin  the  RCT,  as  a  result  other  related  activities  have  been 
delayed  in  order  to  adjust  the  timeline. 

•  Course  of  action  being  taken:  The  PI  and  project  coordinator  are  dedicated  to  getting  the  RCT 
started  in  January  2018.  Therefore,  they  are  scheduling  extra  meetings  and  working  extremely 
hard  to  get  everything  ready  for  the  RCT. 

4.  Ongoing  Problem:  The  online  platform  which  will  host  the  product  during  the  development  phase, 
is  blocked  on  the  VA  network.  This  makes  it  difficult  for  staff  to  work  on  the  project. 

•  Course  of  action  being  taken:  Project  Staff  and  CNDG  have  come  up  with  temporary  solutions 
that  allow  access  to  the  world  when  needed.  Long  term,  the  product  will  be  black  boxed  and 
placed  behind  the  firewall  so  that  internal  DOD  and  VA  personnel  will  be  able  to  access  the 
product. 
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Changes  that  had  a  significant  impact  on  expenditures 

Less  was  expended  on  payroll  during  the  initial  budget  period  due  to  staffing  shortages.  We  have 
since  worked  with  the  DoD  to  carry  forward  those  funds  and  have  applied  them  towards  increased 
staffing  in  an  effort  to  meet  future  milestones. 

Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals, 
biohazards,  and/or  select  agents 

Nothing  to  report. 

Significant  changes  in  use  or  care  of  human  subjects 

Nothing  to  report. 

Significant  changes  in  use  or  care  of  vertebrate  animals. 

Not  applicable. 

Significant  changes  in  use  of  biohazards  and/or  select  agents 

Not  applicable. 
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6.  PRODUCTS 


Books  or  other  non-periodical,  one-time  publications. 

Nothing  to  report. 


Other  publications,  conference  papers,  and  presentations. 

Presentations: 


Shershneva,  Marianna  B.,  Koenig,  Christopher  J.,  Douraghi,  Mathew,  Sabino,  Eilleen  E.,  and  K.  H. 
Seal.  Using  Rapid  Qualitative  Analysis  to  Support  the  Development  and  Implementaion  of  a  Virtual 
World  Training  for  Primary  Care  Providers  on  Caring  for  Veterans  with  Post-traumatic  Stress  Disorder 
Symptoms.  Society  for  Academic  Continuing  Medical 
Education’s  Annual  Meeting.  May  2017.  Scottsdale,  AZ.  (Appendix  F) 


Website(s)  or  other  Internet  site(s) 

A  project  website  focused  on  recruitment  and  dissemination  of  project  information  was  developed 
during  Year  2.  The  website  is  still  private  because  it  is  undergoing  approval  from  the  IRB  but  it  will  go 
live  for  the  start  of  the  RCT  in  January  2018. 

URL:  http://pcpsandptsd.com/ 


Technologies  or  techniques 

Nothing  to  report. 


Inventions,  patent  applications,  and/or  licenses 
Nothing  to  report. 

Other  Products 

There  are  several  products  for  internal  use,  which  facilitate  the  design  and  implementation  of  the 
virtual  world  training: 

•  The  qualitative  team  produced  a  Technical  Report  (Appendix  A)  for  internal  use.  The  report 
details  data  collected  through  semi-structured  interviews  with  PCPs  and  its  relevance  to 
current  research  in  the  field  of  PTSD  treatment,  as  well  as  virtual  world  education.  The 
qualitative  team  produced  a  matrix  for  internal  use  with  Focus  Group  data  analysis  (Appendix 

B) .  The  matrix  will  be  used  to  provide  feedback  to  the  developers  about  changes  that  still  need 
to  be  made  in  the  world 

•  A  storyboard  representing  the  overall  look  and  feel  of  the  virtual  world  environment.  (Appendix 

C) 

•  A  storyboard  for  each  of  the  two  training  Exploratoriums.  (Appendix  D) 
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o  Within  these  storyboards  is  the  framework  for  “Exploratorium”  learning  stations  that 
potentially  can  be  used  as  stand-alone  training  tools. 

•  A  didactic  for  each  of  the  two  training  lectures  (Appendix  E) 

o  These  didactics  could  potentially  be  used  as  stand-alone  training  tools 
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7.  PARTICIPANTS  AND  OTHER  COLLABORATING  ORGANIZATIONS 


What  individuals  have  worked  on  the  project? 

Project  Staff 

Name:  Karen  Seal,  MD,  MPH 
Project  Role:  Principal  Investigator 
Nearest  person  month  worked:  2.4 
Contribution  to  Project:  Unchanged 

Name:  Ryan  McCormick 

Project  Role:  Project  Coordinator 

Nearest  person  month  worked:  3.5 

Contribution  to  Project:  Left  institution  in  August  2017 

Name:  Coleen  Hill 

Project  Role:  Project  Coordinator 

Nearest  person  month  worked:  1 

Contribution  to  Project:  Began  working  on  project  in  September  2017 

Name:  Nicole  R.  McCamish 
Project  Role:  Project  Manager 
Nearest  person  month  worked:  7.2 
Contribution  to  Project:  Unchanged 

Name:  Linda  Abadjian,  PhD 
Project  Role:  Evaluator 
Nearest  person  month  worked:  1 .8 
Contribution  to  Project:  Unchanged 

Name:  Yongmei  Li,  PhD 
Project  Role:  Statistician 
Nearest  person  month  worked:  6 
Contribution  to  Project:  Unchanged 

Name:  Shira  Maguen,  PhD 
Project  Role:  Co-Investigator 
Nearest  person  month  worked:  0.6 

Name:  Thomas  Neylan,  MD 
Project  Role:  Co-Investigator 
Nearest  person  month  worked:  0.3 
Contribution  to  Project:  Unchanged 

Name:  Beth  Cohen,  MD,  MAS 
Project  Role:  Co-Investigator 
Nearest  person  month  worked:  0.6 
Contribution  to  Project:  Unchanged 


Name:  Greg  Reger,  PhD 


Project  Role:  Co-Investigator 
Nearest  person  month  worked:  0.6 
Contribution  to  Project:  Unchanged 

Name:  Jamie  Chang,  PhD 
Project  Role:  Qualitative  Researcher 
Nearest  person  month  worked:  0.15 

Contribution  to  Project:  Began  working  on  project  in  February  2017 

Name:  Natalie  Purcell,  PhD 
Project  Role:  Qualitative  Researcher 
Nearest  person  month  worked:  0.0 

Contribution  to  Project:  Began  working  on  project  in  October  2017 


Vendors 

Name:  Chant  Newall  Development  Group 
Project  Role:  Consultant 
Paid  by  deliverables  per  the  SOW 
Contribution  to  Project:  Unchanged 

Name:  Forefront  Collaborative 
Project  Role:  Consultant 
Paid  hourly  on  a  semi-annual  basis 
Contribution  to  Project:  Unchanged 

Name:  Heyden  Ty 
Project  Role:  Consultant 
Paid  by  deliverables  per  the  SOW 
Contribution  to  Project:  Unchanged 

Has  there  been  a  change  in  the  active  other  support  of  the  PD/PI(s)  or  senior/key 
personnel  since  the  last  reporting  period? 

Natalie  Purcell,  PhD  joined  the  team  in  October  201 7  at  3  calendar  months  to  finish  the  focus  groups 
and  begin  preparations  for  the  standardized  patient  portion  of  the  study. 


What  other  organizations  were  involved  as  partners? 

Nothing  to  report. 


8.  SPECIAL  REPORTING  REQUIREMENTS 
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Gantt  Chart  W81XWH-15-C-0088 


Improving  Access  to  Care  for  Warfighters:  Virtual 
Worlds  Technology  to  Enhance  Primary  Care  Training 

FY  2016 

FY  2017 

FY  2018 

FY  2019 

Q1 

Q2 

Q3 

Q4 

Q1 

Q2 

Q3 

Q4 

Q1 

Q2 

Q3 

Q4 

Q1 

Q2 

Q3 

Q4 

Major  Task  2:  Semi-Structured  interviews 
with  project  stakeholders/key  informants  to 

Anticipated 

N=up 
to  20 

inform  curriculum  content  and  instructional 
design 

Actual 

N= 

11 

Aim  1 

Formative 

Eval/Design 

Major  Task  4:  Independent  review  of  new 
VW  training  using  a  focus  group  -  Recruit 

Anticipated 

N=8 

stakeholders/key  informants  who  were  not 
involved  in  initial  build  recommendations 

Actual 

N=3 

Major  Task  5:  Refinement  of  the  VW 
training  -  Independent  reviewers  for  Focus 
Group 

Anticipated 

N=8 

Actual 

N=5 

Aim  2  RCT 

Major  Task  1 1 :  Recruit  PCPs  from  VA, 

Anticipated 

N=100 

DoD,  and  community 

Actual 

Major  Task  19:  Conduct  Summative 
Analysis  -  Recruit  PCPs  from  each  training 
arm 

Anticipated 

N= 

20 

Aim  3 
Summative 

Actual 

Eval  / 

Disseminate 

Major  Task  19:  Conduct  Summative 

Anticipated 

N= 

40 

Analysis  -  Recruit  stakeholders  from  Aim  1 

Actual 

Updated  10/23/2017 
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“Improving  Access  to  Care  for  Warfighters:  Virtual  Worlds  Technology  to  Enhance  Primary 
Care  Training  in  Posttraumatic  Stress  and  Motivational  Interviewing” 

Log  #JW1 40067  Award  #  W81 XWH-1 5-C-0088 


PI:  Karen  Hope  Seal,  MD  MPH  Org:  Northern  California  Institute  for  Research  and  Education  Award  Amount:  $931 ,538  Directs  +  $496,510  F&A 


Study  Aims 

Veterans  present  to  primary  care  providers  (PCPs)  with  posttraumatic  stress  (PTS)  symptoms  because  many 
are  resistant  to  specialty  mental  health  care.  Most  PCPs  have  not  been  trained  to  assess  for  and  manage 
symptoms  of  PTS  or  motivate  Veterans  to  engage  in  treatment.  This  can  result  in  missed  opportunities  to 
intervene  to  prevent  chronic  mental  and  physical  health  problems.  We  propose  to: 

(1)  Iteratively  design  a  new  web-based  PTS  and  Motivational  Interviewing  training  for 
PCPs  using  Virtual  World  technology  to  enhance  interactivity. 

(2)  Add  a  more  robust  evaluation  including  a  randomized  control  trial  for  more  clinically 
valid  outcome  measurement. 

(3)  Conduct  a  summative  evaluation  to  inform  national  “scale-up”  dissemination  and 
implementation. 

The  proposed  project  is  aligned  with  the  needs  of  JPC  5  (Psychological  Health  and  Resilience)  and  will 
produce  a  deliverable  that  will  improve  access  to  quality  clinical  care  for  our  warfighters  suffering  with  PTS. 

Approach 

We  are  using  mixed  qualitative  and  quantitative  observational  and  experimental  methods  to  conduct  a 

4-vear  effectiveness-implementation  randomized  controlled  trial  (RCTT  in  which  project  stakeholders 

participate  from  start  to  finish.  A  formative  evaluation  consisting  of  focus  groups  and  semi-structured 
interviews  captures  stakeholder  input  in  how  we  can  best  design  and  implement  the  Virtual  World  (VW) 
training.  We  will  then  conduct  an  RCT  to  compare  the  new  VW  training  to  our  prior  online  PTS  training.  Pre- 
/post-  and  follow-up  standardized  patient  interviews,  provider  self-report  measures,  and  patient  outcomes  will 
be  compared  between  groups.  A  summative  evaluation  will  solicit  feedback  of  PCP  participants  and 
stakeholders  to  expedite  dissemination  of  the  new  VW  training. 

Timeline  and  Cost 

Projected  Period  of  Performance  and  Proposed  Direct  Costs  Per  Year 


RCT 

Aim  3  Summative 
Eval/Disseminate 

Estimated  Costs/Year  495,224.05  436,314.56  310,165.91  312,640.78 

($) 

Total  Direct:  $931,538  Directs  +  $496,510  F&A 

Period  of  Performance:  09/25/2016-09/24/2017 
Updated:  10/23/2017 


Goals/Milestones 


Major  Task  1 :  Obtain  local  IRB  and  VA  R&D  and  HRPO  Approvals 

Major  Task  2:  Semi-Structured  interviews  with  project  stakeholders/key  informants  to 

inform  curriculum  content  and  instructional  design 

Major  Task  3:  Begin  Virtual  World  (VW)  build 

Major  Task  4:  Independent  review  of  new  VW  training  using  a  focus  group 
Major  Task  5:  Refinement  of  the  VW  training 

Major  Task  6:  Refinement  of  prior  online  training  (Control)  to  make  it  a  more  apt 
comparison  for  RCT. 

Major  Task  7:  Obtain  CME  accreditation  for  the  trainings 

Major  Task  8:  Develop  self-report  measure  battery  to  assess  relative  change  in  self- 
reported  PTS-related  knowledge  and  clinical  skills  self-efficacy. 

Major  Task  9:  Develop  telephone  Standardized  Patient  (SP)  Interview  to  assess  relative 
change  in  PTS-related  competency  (Primary  Outcome)  and  Ml  skills  to  assess  and  manage 
PTS  symptoms  and  to  motivate  engagement  in  care. 

Major  Task  10:  Recruitment/Eligibility  Screening 

Major  Task  11:  Recruit  PCPs  from  VA,  DoD,  and  community 

Budget  Expenditure  to  Date 
Projected  Expenditure:  $436,314.56 
Actual  Expenditure:  $397,222.00 


9.  APPENDIX:  N/A 


24 


